Skip to content

feat(rules): semantic rules with closed verdicts, Jev and Anthropic judges - #151

Merged
mindsocket merged 22 commits into
mainfrom
spike-114a-builtin-provider
Oct 15, 2026
Merged

mindsocket merged 22 commits into
mainfrom
spike-114a-builtin-provider

Conversation

@mindsocket

@mindsocket mindsocket commented Oct 13, 2026 •

Copy link
Copy Markdown
Owner

Implements the core of #114. Left open for the remaining questions there: caching, retry and concurrency, and the evals layer (#116).

Builds on cascade gating (#148, merged). Supersedes #149, which GitHub auto-closed when its stacked base branch was deleted.

What

Semantic rules (mode: semantic) judged by an LLM or a decision model, running inside cascade gating. Opt-in per run and inert otherwise: nothing is sent anywhere unless a run passes --semantic or --judge <provider> (or an API caller passes a judge). Config cannot turn judging on. A schema with no semantic rules behaves exactly as before.

  • Closed verdicts. A rule declares verdicts; default pass/fail. A verdict is a description for the judge, or {description, guidance} where guidance is shown to the user when it fires (never sent to the judge). Judges return a verdict plus optional confidence, probabilities and evidence.

  • Verdict is classification, severity is consequence. severity is a level, or for semantic rules a map {verdict: error|warning|info|ignore}. Verdicts left out of the map pass; ignore means the rule ran and the verdict is recorded but not reported. Required when custom verdicts are declared (there is no failOn). Gating follows the verdict's severity.

  • when. An optional JSONata applicability test on any per-node rule, evaluated before the rule (so before any judge call). Where it is not true the rule is skipped silently, neither passed nor gated.

  • Model providers, then the judge. Config separates how to reach a model from which one judges. Top-level modelProviders names each provider (type, model, endpoint, apiKeyEnv). judge.provider picks the judge, at the top level or overridden per space (for example, a private space on a local model). The provider is never inferred from API keys in the environment, so a key set for another tool is never spent unless the user chose that provider. There are no env overrides: config says what, flags override per run, env holds secrets only. A misconfigured judge is a plain CLI error, raised before any request and only when a semantic rule is to be judged.

  • Rules don't choose a judge. That is config, not schema, so schemas stay portable across configs. --judge <provider> picks a different configured provider for one run (implies --semantic). Config-level overrides for classes of rules are a possible follow-up.

  • Providers (src/judge/provider/): anthropic (default claude-haiku-5-5), openai (any OpenAI-compatible chat endpoint, key optional for local servers), jev (TypeSafe decision model through OpenRouter's alpha decisions API, OPENROUTER_API_KEY), fake for tests. All are plain fetch calls; no SDK dependencies.

  • Per-node batching. All of a node's semantic rules go to the judge in one call. Chat providers make one request per rule inside it; Jev answers them in one request.

  • Gating. Expression rules run first, then the judge. Semantic rules are skipped on gated nodes and gate their subtree per gatesDescendants. A judge error is reported as a violation but never gates.

  • validate-file is cheap. Gating is computed over the whole space, but the judge is called only for the file and for its ancestors' gate-capable semantic rules (top-down, stopping at the first gate): at most one call per ancestor level plus one.

  • --rule-filter dim=values (mode, category, severity, type, id; repeatable; a verdict map matches any severity it can report) and --judge <provider> on validate and validate-file.

  • Rule contract. check is required unless mode: semantic; semantic rules cannot declare check or scope: global; verdicts, rubric, judge and a severity map are semantic-only; when cannot combine with scope: global. sctx schemas show lists a rule's mode, severities, judge and when.

  • Docs. docs/rules.md (now "Rules") owns the shared ideas: both kinds of rule, violations, when and --rule-filter. New docs/semantic-rules.md (sctx docs semantic-rules) covers only what differs, starting with a quick start. Judge fields are in docs/config.md and terms in docs/concepts.md.

  • No new public API. structured-context/api exports are unchanged. validateSpace and validateFile gain optional judge (a resolved judge: present means judge) and ruleFilter options, and RuleViolation gains optional evidence, verdict and guidance. Judge providers, the factory and the judge types stay internal: building a judge is a core and CLI concern, and --semantic / --judge <provider> are resolved into a judge at the CLI.

Not in this PR

  • No retry, concurrency limit or caching.
  • Rule scope beyond the node itself (ancestors, siblings, input format): #152.
  • Claude plugin skills, hooks and commands for semantic rules: #153.
  • Config-level overrides of rule or verdict severity (the lint-style "turn off this verdict for my space"), disabling a rule, and ignoring results by node: a rule's severity map is shaped to be overridden by config later. See #92.
  • Layered judging (a decision model decides, an LLM explains): a profile concept is reserved in the shape but not built; guidance covers coaching for now.
  • includeGated is still one switch for both rule kinds (it gates deterministic and semantic rules alike). Splitting them, so an editor can un-gate cheap rules while judged ones stay gated, is a follow-up.
  • Hierarchy gating applies to the main hierarchy only.
  • Other decision-model APIs are not supported. OpenAI has a Decisions API in public beta (POST /v1/decisions, input plus questions) with a different wire shape from OpenRouter/TypeSafe's (state plus keyed questions); both are candidates for further providers behind the same interface.

Testing

660 unit tests (bun run test); typecheck and biome are clean. New tests cover verdict resolution, prompt/response parsing, each provider (HTTP mocked), provider and judge config (including that no provider is ever inferred from the environment, and the space and run overrides), rule contract, rule filter, batching, gating interaction, which nodes validate-file judges, verdict severity/ignore/guidance and per-verdict gating, when, and the CLI --judge wiring.

  • Live judge tests. bun run test:judge sets SCTX_LIVE_JUDGE_TESTS=1 and runs judge-test/ against a fixture space with each provider whose API key is set (OPENROUTER_API_KEY, ANTHROPIC_API_KEY, OPENAI_API_KEY). Without the variable they skip, so bun test stays offline. Jev and OpenAI pass live; Anthropic has not run yet, because no key was available. The OpenRouter decisions endpoint is still alpha.
  • Expression errors are violations. A check or when that cannot be evaluated (e.g. a syntax error) is reported as an error violation (Rule error: ...) instead of silently passing; it does not gate descendants.

Comment thread src/config.ts Resolved
Comment thread src/validate.ts Resolved
@claude

claude Bot commented Oct 13, 2026

Copy link
Copy Markdown

Code review

No issues found. Checked for bugs and CLAUDE.md compliance.

@mindsocket mindsocket left a comment

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The docs leave a bit to be desired in terms of content design and approachability. Keep focused, move more toward ASD-STE100, refer to other docs/sections, (it's ok to repeat small tidbits when necessary and non-obvious), consider information architecture (eg rules doc should capture key ideas, and semantic rules should point to them and take as read instead of randomly tossing things in), introduce concepts once and properly before sections that need them described.

Comment thread docs/semantic-rules.md Outdated
@@ -0,0 +1,103 @@
# Semantic Rules

A semantic rule is a rule a model judges instead of a JSONata expression: "is this opportunity framed in the customer's voice?" rather than "does this field exist?". Semantic rules are **opt-in**. Nothing is sent to a model unless you pass `--semantic` (or enable it in config), and a space with no semantic rules is unaffected.

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is too verbose. No need to mention jsonata, "rather than", 2 long sentences for how to run it when 1 short statement will do. Apply same thinking throughout updated docs.

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Rewritten in 6921788. The intro is now two short sentences: what a semantic rule is and when to use one. The rest of the doc gets the same treatment.

Comment thread docs/semantic-rules.md Outdated
}
```

2. **Choose a judge** in `config.json`. The judge `type` is always explicit; keys in your environment are never picked up unless you name the type:

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is confusing language. Is this a quickstart or not? Just say "use built in anthropic type and it will pick up ANTHROPIC_API_KEY" or something

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The quick start now says: set OPENROUTER_API_KEY, add { "judge": { "type": "jev" } }. One sentence gives Claude as the alternative ("type": "anthropic" with ANTHROPIC_API_KEY).

Comment thread docs/semantic-rules.md Outdated

with `ANTHROPIC_API_KEY` set. See [Judge configuration](config.md#judge-semantic-rules) for every field.

3. **Run it**: `sctx validate <space> --semantic`. To turn it on by default, set `semantic: true` (or `judge.autoRun`) in config; `--no-semantic` overrides that for a single run.

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Again, not quickstart material.

Aside, why are there 2 ways to turn it on (there could be a good reason), and where do these things go. These random bits of assumed knowledge suggest poorly thought through content design.

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There was no good reason for two switches. judge.autoRun duplicated semantic, so I removed it from the code and docs. The quick start just runs with --semantic. A new section, "Turning semantic rules on", covers --semantic, "semantic": true (top level or per space, linked to config.md) and --no-semantic.

Comment thread docs/semantic-rules.md Outdated

### Verdicts, severity and guidance

The judge must choose exactly one verdict. A closed set keeps results comparable between runs and maps directly onto decision models, which return a choice with confidence and probabilities rather than prose.

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

blah blah.

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cut. The verdicts section now says the judge gives one verdict per rule and node, from a fixed set, and that a fixed set makes results comparable between runs.

Comment thread docs/semantic-rules.md Outdated

The judge must choose exactly one verdict. A closed set keeps results comparable between runs and maps directly onto decision models, which return a choice with confidence and probabilities rather than prose.

- **A verdict is a classification, severity is the consequence.** `severity` as a map decides what each verdict means for the user: `error`, `warning` or `info`. A verdict left out of the map passes, and `ignore` states that explicitly: the rule ran and the verdict was recorded, but nothing is reported. At least one verdict must pass.

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"passes" isn't well defined, assumes that "passing" a rule is a thing, and I'm not sure that language is uniquitous and clear.

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed. rules.md now defines a violation under Severity: a rule either reports a violation (with a severity) or reports nothing. The semantic docs say a verdict "reports a violation" or "reports no violation", and never "passes".

Comment thread docs/semantic-rules.md Outdated
"when": "current.status = 'ready' and $toMillis(current.due) - $millis() < 604800000" }
```

`when` works on expression rules too. It saves money on semantic rules because the cheap test runs first.

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

blah blah.

Better to document "when" once in rules, and how it helps optimise rule running and semantic rule costs. Then this doc needs little more.

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done. rules.md documents when once, including that it costs almost nothing and saves a model call on each node it excludes from a semantic rule. semantic-rules.md only links to it. --rule-filter also moved to rules.md, because it applies to all rules.

Comment thread docs/semantic-rules.md Outdated
| `openai` | Any OpenAI-compatible chat endpoint: OpenAI, Ollama, vLLM, Groq, OpenRouter chat models | Set `endpoint` for non-OpenAI servers; the key is optional for local ones |
| `jev` | TypeSafe's Jev decision model | Closed-set answers with calibrated probabilities, one request per node. Uses an OpenRouter key |

Prefer a decision model when you can: results are repeatable, cheap, and come with a confidence. Use a chat LLM when the rubric needs open reasoning or evidence text.

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The preferred model sure is buried here. Put it in the quickstart.

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Jev is now step 2 of the quick start, as the recommended judge. Choosing a judge lists it first. In f6b3e04 I verified Jev against the live OpenRouter endpoint, and bun run test:judge now covers it.

Comment thread docs/semantic-rules.md Outdated
|---|---|---|
| `anthropic` | Claude models | Default model `claude-haiku-5-5` |
| `openai` | Any OpenAI-compatible chat endpoint: OpenAI, Ollama, vLLM, Groq, OpenRouter chat models | Set `endpoint` for non-OpenAI servers; the key is optional for local ones |
| `jev` | TypeSafe's Jev decision model | Closed-set answers with calibrated probabilities, one request per node. Uses an OpenRouter key |

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

To check: are error messages helpful when someone's set jev but not provided an OPENROUTER_API_KEY

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Checked from the CLI with OPENROUTER_API_KEY unset. It exits 1 with: Error: Judge "default" (jev) has no API key. Set OPENROUTER_API_KEY in the environment, or "apiKey" / "apiKeyEnv" on the provider.

- **`references/schema-design.md`** — process for designing a schema from existing content

For CLI and config reference, use `sctx docs <topic>` (topics: `concepts`, `config`, `schema`, `rules`).
For CLI and config reference, use `sctx docs <topic>` (topics: `concepts`, `config`, `schema`, `rules`, `semantic`).

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

rename doc to semantic-rules so this makes more sense

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Renamed: sctx docs semantic-rules. I updated SKILL.md, the README, rules.md, schema-authoring.md and the docs command help to match.

Comment thread src/judge/factory.ts Outdated
// OpenAI-compatible endpoints (Ollama, vLLM, ...) may need no key.
if (!apiKey && type !== 'openai') {
throw new JudgeConfigError(
`Judge "${name}" (${type}) has no API key. Set ${keyEnv} in the environment, or "apiKey" / "apiKeyEnv" on the provider.`,

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks like this is the friendly error message. 👍

@mindsocket

Copy link
Copy Markdown
Owner Author

Re: the review on docs content design (6921788):

  • Information architecture. rules.md (now titled "Rules") owns the shared ideas. It introduces both kinds of rule first, then defines a violation, when (with its cost benefit) and --rule-filter. semantic-rules.md takes these as read, links to them, and covers only what differs: the quick start, turning semantic rules on, rule fields, verdicts, severity, how judging works and choosing a judge.
  • Style. The docs now use short, declarative sentences in active voice, one idea per sentence, and terms used consistently (for example "reports a violation" instead of "passes"). config.md's judge section and the concepts entry are trimmed the same way.
  • Concepts are introduced before use. Gating and when are linked where they first appear, not re-explained.

Also in this push (f6b3e04):

  • Opt-in live judge tests, run with bun run test:judge.
  • Two bugs those tests found: validate ignored --judge, and validate --json left out gatedRules.

Comment thread docs/config.md Outdated
| `apiKeyEnv` | The environment variable that holds the key. Defaults: `OPENROUTER_API_KEY` (jev), `ANTHROPIC_API_KEY`, `OPENAI_API_KEY`. A local `openai` endpoint can need no key. |
| `apiKey` | The key itself. Use `apiKeyEnv` instead, so that keys stay out of config files. |

sctx creates a provider only when it uses it, so a provider that is not used needs no key. The environment holds only keys: no environment variable selects or configures a provider.

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Still overexplaining. Also, don't use sctx shorthand in documentation. sctx is the cli command and informal shorthand, structured-context is the proper project name.

Suggested: "The validation engine creates a provider on when it uses it."

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in 5c02201: the section is now "Model providers and judge", merged per the next comment, and prose in docs uses structured-context (the Documentation Principles section in AGENTS.md now records this).

Comment thread docs/config.md Resolved
Comment thread docs/config.md Resolved
Comment thread docs/rules.md Outdated
- An **expression rule** (the default) is a JSONata `check` that must return `true`.
- A **semantic rule** asks a model to judge a node. See [Semantic rules](semantic-rules.md).

This page applies to both kinds, unless it says otherwise.

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Redundant?

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cut in 5c02201.

Comment thread docs/rules.md Outdated
| `id` | yes | Unique rule identifier |
| `category` | yes | `validation` \| `coherence` \| `workflow` \| `best-practice` |
| `severity` | no | `error` \| `warning` \| `info`; overrides the category default (see [Severity](#severity)) |
| `severity` | no | `error` \| `warning` \| `info`; overrides the category default (see [Severity](#severity)). A semantic rule can map verdicts to severities |

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If this is covered in severity section and/or semantic-rules, don't add random snippets about semantic rules here. This applies generally as a principle too, unless the callout is of critical importance in context.

Comment thread docs/semantic-rules.md Resolved
Comment thread docs/semantic-rules.md Outdated
| `anthropic` | Claude. Default model: `claude-haiku-5-5` | `ANTHROPIC_API_KEY` |
| `openai` | Any OpenAI-compatible chat endpoint, for example OpenAI, Ollama or vLLM. Set `endpoint` for servers other than OpenAI. Local servers do not need a key | `OPENAI_API_KEY` |

sctx uses only the provider that you configure. It does not choose a provider from the keys in your environment. See [Model providers](config.md#model-providers) and [Judge](config.md#judge-semantic-rules) for all fields.

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

s/configure/specify/ might be better. You can specify "jev" without configuring it (with a key) per-se.

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in 5c02201.

@@ -0,0 +1,5 @@
---
type: opportunity

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Consider converting this whole space fixture into a single json or jsonl of spacenodes, or whatever the judge test needs, unless the test is necessarily to exercise more of the parse/validate pipeline.

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The test's job is to exercise the judge end-to-end through validation (gating decides whether the judge runs at all), so some pipeline coverage is intentional. Still worth simplifying; parked with the other follow-ups in #155-#157 rather than in this PR.

Comment thread judge-test/judges.test.ts Outdated
};

function validate(...args: string[]): { exitCode: number; output: Output } {
const result = Bun.spawnSync(['bun', 'run', 'src/index.ts', 'validate', 'judge-test', '--json', ...args], {

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we need to do quite the extensive dance with a process call and full parse just because of environment? Could this test be simplified?

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same answer as the fixture comment: the subprocess-and-parse setup exists so the test exercises the same gating path a real run takes. Simplification parked with #155-#157 follow-ups.

Comment thread config.example.json Outdated
],
"schema": "/path/to/custom/schemas/my-general-schema.json"
"schema": "/path/to/custom/schemas/my-general-schema.json",
"modelProviders": {

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We discussed not having default config accidentally enabling model calls, but perhaps requiring --semantic is already enough.

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed with the musing: kept explicit modelProviders, but config.example.json now uses the shorthand ("judge": { "type": "jev" }, 5c02201), so the example no longer implies that default config enables model calls — a run still has to ask with --semantic/--judge.

Semantic rules (mode: semantic) declare a closed verdict set (verdicts, failOn)
and are judged per node, all of a node's rules in one provider call. Judges:
Anthropic (default claude-haiku-5-5), OpenAI-compatible, Jev via OpenRouter
decisions API, fake. Semantic rules run inside cascade gating (#115) after
deterministic rules and only with --semantic. --rule-filter narrows rules by
mode, category, severity, type or id.
…es, accept jev in config

Addresses review on the semantic rules PR.
Per the definition of done: config.md (judge/semantic keys), concepts.md
(semantic rule, judge, verdict; gating no longer 'future'), api.md (judging
section, option lists), AGENTS.md (src/judge, rule-filter), config example,
and the remaining src/api.ts exports. Adds a config test for provider jev.
…e/provider, split semantic docs, trim test fixtures
…expression errors as violations

- judge config uses `type` in the flat and named forms; `use` becomes `default`
- remove SCTX_JUDGE_* overrides (config, flags and apiKeyEnv cover them); document where a setting lives
- a check or when that cannot be evaluated is an error violation, not a silent pass
- okf: stale_after ordering is a validation rule; drop redundant explicit severities
…Run; rename docs topic to semantic-rules

rules.md now owns the shared ideas (violation, when, rule filtering) and
semantic-rules.md covers only what differs, in shorter sentences. Quick
start leads with Jev. judge.autoRun duplicated the semantic config key,
so semantic is now the only config switch. sctx docs semantic becomes
sctx docs semantic-rules.
… and gatedRules missing from JSON

bun run test:judge sets SCTX_LIVE_JUDGE_TESTS=1 and runs judge-test/
against a fixture space with each provider whose API key is set.
Without the variable the tests skip, so bun test stays offline.

The validate action did not pass --judge through, so naming a judge
there ran no semantic rules. validate --json also left out gatedRules,
which the docs list.
… API; guard API surface in AGENTS.md

No consumer needs the judge providers, factory, judge types or rule
filter, so src/api.ts and docs/api.md return to their previous scope.
AGENTS.md now requires explicit agreement before any export is added.
…types

Violations and coverage gates already report what users act on, so the
per-evaluation and per-gated-rule records go, with their JSON fields and
CLI summary lines. The judge input node is a Pick of SpaceNode, the rule
spec no longer carries failOn (severity stays out of the judge's input),
and JudgeConfig extends the provider definition with a real type union.
… for semantic rules

Config now separates how to reach a model (top-level modelProviders) from
which one judges (judge.provider, top level, with a space override). Config
no longer turns semantic rules on: --semantic or --judge <provider> (or the
API options) ask on each run, so --no-semantic and the hooks' pin go.
Rules no longer name a judge: that is config, not schema. The judge is built
once per run, and only when a semantic rule is to be judged.
… to judge

validateSpace and validateFile take an optional resolved judge instead of
semantic, a judge name and an injected provider: a judge means judge.
--semantic and --judge are CLI concerns, turned into a judge by
judgeForRun. validateRules follows the same shape internally.
A judge names either a bundled provider type ({ "type": "jev" }) or an
entry in modelProviders ({ "provider": "..." }); a modelProviders entry
wins over a bundled type of the same name. No judge in config means no
model runs, and a run still asks with --semantic or --judge.

Docs: merge config.md's provider and judge sections, adopt a
Documentation Principles section in AGENTS.md (one idea lives in one
place; document what it does; structured-context in prose; front-load
the happy path), and trim rules.md per review.
mindsocket and others added 6 commits October 15, 2026 23:34
judge.provider names a provider: a modelProviders entry first, else a
bundled provider (jev, anthropic, openai). A user entry replaces a
bundled one of the same name, keeping type defaults for fields it leaves
out. The fake provider stays test-only. The judge-test fixture space is
gone: the live provider test now judges one inline node directly
(SCTX_LIVE_JUDGE_TESTS=1, per-provider key check), and pipeline coverage
stays with the existing fake-judge tests.
Provider definitions name a wire API (provider.api: openai-completions,
anthropic-messages, openrouter-decisions, typesafe-systemone) instead of
a model-flavoured type. The two decisions APIs share one implementation.

Hydration happens in the factory, not the providers: explicit value >
named env var (set-but-empty is an error) > API default. Provider
implementations read no environment; only the key has a per-API default
variable. The bundled catalogue now carries vendor bundles:
openrouter-jev (typesafe/jev-latest), typesafe-jev, anthropic,
anthropic-haiku-5-5, anthropic-env, openai, openai-env, openrouter.

New sctx providers lists bundled and configured providers with their
effective config, keys never shown.
…ctx providers

The openrouter bundle spoke openai-completions with no endpoint, so its
key would be sent to api.openai.com. It now pins
https://openrouter.ai/api/v1. sctx providers shows each provider's
effective endpoint, and bundle defaults are pinned by tests.
…setup

- new openai-decisions API and OpenAIDecisionsJudgeProvider (one request
  per node, choice questions from the rule's verdict set; verified live
  against api.openai.com/v1/decisions)
- new bundles: openai-decisions, openrouter-openai-decisions; openrouter
  chat bundle renamed to openrouter-openai
- model constants live in the provider modules; the factory imports them
  (recommended models per vendor are this project's judgement; pinned
  bundles keep their literal)
- OpenRouter model slug verified: ~typesafe/jev-latest resolves on the
  decisions endpoint; typesafe/jev-latest does not
- docs: bundled-provider tables replaced by "sctx providers" plus the
  wire-API table and naming conventions
- bun run test:models smoke-checks each bundled model resolves (keyless
  where the endpoint allows; skip per provider without a key)

Co-authored-by: Claude <noreply@anthropic.com>
@mindsocket
mindsocket force-pushed the spike-114a-builtin-provider branch from d8ceb90 to d997be9 Compare October 15, 2026 12:38
@mindsocket
mindsocket merged commit d997be9 into main Oct 15, 2026
1 of 2 checks passed
@mindsocket
mindsocket deleted the spike-114a-builtin-provider branch October 15, 2026 12:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant